<html>
<head><meta charset="utf-8"><title>complications in wrapping AsyncRead/AsyncWrite · wg-async-foundations · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/index.html">wg-async-foundations</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html">complications in wrapping AsyncRead/AsyncWrite</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="232163260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232163260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ron Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232163260">(Mar 28 2021 at 12:16)</a>:</h4>
<p>Hi all, I'm new here, so please apologies if this has already been discussed. I've been following the async vision process from the sideline - thank you so much for investing in this. I was curious to hear if you've been thinking about the future of AsyncWrite/AsyncRead (+it's siblings) and defining them in terms of "regular" async functions instead of what we have today which return <code>Poll</code>?</p>



<a name="232163326"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232163326" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ron Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232163326">(Mar 28 2021 at 12:18)</a>:</h4>
<p>I know that async traits require a crate today so that's a complication</p>



<a name="232163453"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232163453" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ron Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232163453">(Mar 28 2021 at 12:21)</a>:</h4>
<p>In my experience (maybe I missed something) any stateful wrapper around an AsyncRead/AsyncWrite must manage their state manually. An example is <a href="https://github.com/Nemo157/async-compression/blob/prīmum/src/tokio_03/bufread/generic/decoder.rs#L61-L120">async-compression</a>. It would be great if we were able to use all the generator magic that the compiler helps us with in regular <code>async</code> functions and simply write the wrapping read method as <code>async fn read(..) -&gt; Result&lt;..&gt;</code></p>



<a name="232170227"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232170227" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Yoshua Wuyts <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232170227">(Mar 28 2021 at 14:26)</a>:</h4>
<p><span class="user-mention" data-user-id="386909">@Ron Cohen</span> That's a great question! I have an unpblished post that attempts to cover this. My current thinking is that if we could implement <code>impl IntoStream for AsyncBufRead</code> (or similar), it would make it easy real easy to then create custom transforms using generator functions.</p>



<a name="232170248"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232170248" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Yoshua Wuyts <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232170248">(Mar 28 2021 at 14:27)</a>:</h4>
<p>Unfortunately I don't think it's likely we'll see easier ways to define <code>AsyncRead</code> / <code>AsyncWrite</code> <em>directly</em>. But I think we can work around that by making it easier to create pipelines that work with existing <code>AsyncRead</code> / <code>AsyncWrite</code> impls. That would make implementing your own <code>AsyncRead</code> / <code>AsyncWrite</code> a rare thing to do, and make the majority use case much simpler.</p>



<a name="232170429"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232170429" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Yoshua Wuyts <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232170429">(Mar 28 2021 at 14:31)</a>:</h4>
<p>(Somewhat analogous to how on Node.js almost all userland stream impls used to rely on <code>through2</code>, with only a minority of streams actually being implemented in terms of the prototype interfaces)</p>



<a name="232170439"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232170439" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ron Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232170439">(Mar 28 2021 at 14:31)</a>:</h4>
<p>Right. Wrapping streams is definitely easier</p>



<a name="232170649"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232170649" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Yoshua Wuyts <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232170649">(Mar 28 2021 at 14:34)</a>:</h4>
<p>Not sure if there's a user story for this already though -- <em>"Writing your own AsyncRead/AsyncWrite today is an involved process"</em> would def be a good one to track. Maybe someone who's been more active recently can answer whether it's being tracked.</p>



<a name="232170763"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232170763" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ron Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232170763">(Mar 28 2021 at 14:37)</a>:</h4>
<p>here's an example I wrote up: <a href="https://gist.github.com/roncohen/a81d28b3db17766491dfd78864d549c8">https://gist.github.com/roncohen/a81d28b3db17766491dfd78864d549c8</a><br>
in this example the state tracked is very simple, but it can quickly become unwieldy in my experience</p>



<a name="232170958"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232170958" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ron Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232170958">(Mar 28 2021 at 14:40)</a>:</h4>
<p>happy to contribute a user story like that if it doesn't already exist</p>



<a name="232191362"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232191362" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthias247 <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232191362">(Mar 28 2021 at 20:34)</a>:</h4>
<p>If all the code is under your control and you don't to interact with tokio/futures/etc traits, just define your own IO traits on top of <code>async_trait</code>. That makes things certainly a lot easier to use and compose. The drawback is the additional memory allocation per async operation, but maybe it's not an issue for you</p>



<a name="232191505"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232191505" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthias247 <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232191505">(Mar 28 2021 at 20:36)</a>:</h4>
<p>Building all the traits on top of <code>async fn</code>s instead of having more low level combinators was a very very old proposal at <a href="https://github.com/rust-lang/futures-rs/issues/1365">https://github.com/rust-lang/futures-rs/issues/1365</a>, but didn't see alot of traction due to concerns with boxing, missing GATs and the potential for losing <code>dyn Trait</code> (there is no solution which currently avoids all of the issues at once).</p>



<a name="232191710"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232191710" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthias247 <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232191710">(Mar 28 2021 at 20:40)</a>:</h4>
<p><span class="user-mention" data-user-id="211722">@Yoshua Wuyts</span> I don't think node is a good precedent here. The whole stream definition and combinator story was done before async/await arrived there. I'm pretty sure that if node had been written after async/await in JS was standarized streams would just be defined as interfaces containing an <code>async fn</code>.  I think the newer <code>ReadableStream/WriteableStream</code> specifications already go in that direction.</p>



<a name="232196362"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232196362" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232196362">(Mar 28 2021 at 22:01)</a>:</h4>
<p><span class="user-mention" data-user-id="386909">@Ron Cohen</span> the <a href="https://fasterthanli.me/articles/pin-and-suffering">recent "pin and suffering" blog post</a> by fasterthanlime actually covered many details here. I would love to see some status quo stories on this topic, for sure -- I was thiking of trying to turn that post into one, if needed</p>



<a name="232197193"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232197193" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Doc Jones <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232197193">(Mar 28 2021 at 22:16)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116009">nikomatsakis</span> <a href="#narrow/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite/near/232196362">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="386909">Ron Cohen</span> the <a href="https://fasterthanli.me/articles/pin-and-suffering">recent "pin and suffering" blog post</a> by fasterthanlime actually covered many details here. I would love to see some status quo stories on this topic, for sure -- I was thiking of trying to turn that post into one, if needed</p>
</blockquote>
<p><a href="#narrow/stream/187312-wg-async-foundations/topic/article.3A.20pin.20and.20suffering/near/232164903">https://rust-lang.zulipchat.com/#narrow/stream/187312-wg-async-foundations/topic/article.3A.20pin.20and.20suffering/near/232164903</a></p>



<a name="232207123"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232207123" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Doc Jones <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232207123">(Mar 29 2021 at 01:27)</a>:</h4>
<p>(deleted)</p>



<a name="232224761"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/complications%20in%20wrapping%20AsyncRead/AsyncWrite/near/232224761" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Ron Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/complications.20in.20wrapping.20AsyncRead.2FAsyncWrite.html#232224761">(Mar 29 2021 at 06:43)</a>:</h4>
<p>thanks for the links <span class="user-mention" data-user-id="204219">@Matthias247</span> and <span class="user-mention" data-user-id="116009">@nikomatsakis</span>. Between them they illustrate the issue quite well. Let me know if I can help write up the status quo story.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>